Information Seeking Spoken Dialogue Systems- Part II: Multimodal Dialogue

In this paper, the task and user interface modules of a multimodal dialogue system development platform are presented. The main goal of this work is to provide a simple, application-independent solution to the problem of multimodal dialogue design for information seeking applications. The proposed system architecture clearly separates the task and interface components of the system. A task manager is designed and implemented that consists of two main sub-modules: the electronic form module that handles the list of attributes that have to be instantiated by the user, and the agenda module that contains the sequence of user and system tasks. Both the electronic forms and the agenda can be dynamically updated by the user. Next a spoken dialogue module is designed that implements the speech interface for the task manager. The dialogue manager can handle complex error correction and clarification user input, building on the semantics and pragmatic modules presented in [4]. The spoken dialogue system is evaluated for a travel reservation task of the DARPA Communicator research program and shown to yield over 90% task completion and good performance for both objective and subjective evaluation metrics. Finally, a multimodal dialogue system which combines graphical and speech interfaces, is designed, implemented and evaluated. Minor modifications to the unimodal semantic and pragmatic modules were required to build the multimodal system. It is shown that the multimodal system significantly outperforms the unimodal speech-only system both in terms of efficiency (task success and time to completion) and user satisfaction for a travel reservation task. EDICS Category: 3-DIAL 3-MODA 3-INTF Alexandros Potamianos is with the Dept. of Electronics & Computer Engineering, Technical Univ. of Crete, Chania 73100, Greece; email: [email protected]; tel:+30-28210-37221; fax:+30-28210-37542. Eric Fosler-Lussier is with the Dept. of Computer Science and Engineering, The Ohio State University, Columbus, OH 43210, USA; email: [email protected]; tel: +1-614-292-4890; fax: +1-614-292-2911. Egbert Ammicht is with Bell Labs, Lucent Technologies, Whippany, NJ 07981, USA; email: [email protected]; tel: +1-973-386-7139. Manolis Perakakis is with the Dept. of Electronics & Computer Engineering, Technical Univ. of Crete, Chania 73100, Greece; email: [email protected]; tel:+30-28210-37368; fax:+30-28210-37542. Some of this work was performed while Alexandros Potamianos and Eric Fosler-Lussier were with Bell Labs, Lucent Technologies, Murray Hill, NJ 07974, USA. TRANS. ON MULTIMEDIA, VOL.X, NO.XX 2 Information Seeking Spoken Dialogue Systems – Part II: Multimodal Dialogue